Approximation of polynomials by a neural network having rather a small number of units

Abstract–We remark that most neural approximation theorems may not reflect the way of realizing approximation by actual neural networks, and propose an idea that neural networks generally realize approximation by surface-fitting methods, showing versatility of a linear sum of a few basis functions. When the approximation is realized by adjusting parameters of a linear sum so as to fit its surface to the target surface, we call it an approximation by surface-fitting method. This method is characterized by a small number of basis functions in the linear sum, contradicting most neural approximation theorems which postulate availability of arbitrarily many nonlinear units. By this method, any polynomial in several variables can be approximated on compact sets and the approximation can be extended to derivatives. If the surface of the basis function is rich in variety, the linear sum can approximate a wide variety of surfaces. Several examples are illustrated.